Multi-standard video image capture device using a single CMOS image sensor

ABSTRACT

A video image capture device includes an image sensor including an two-dimensional array of pixel elements overlaid with a pattern of color filters and having a vertical resolution different from the vertical resolutions specified for a group of video formats, a frame buffer for storing digital pixel data outputted by the image sensor, and an interpolator module for interpolating the digital pixel data to generate video data in at least three color planes and having a vertical resolution corresponding to a video format selected from the group of video formats. In one embodiment, the group of video formats includes the NTSC and PAL video formats. The vertical resolution of the image sensor has a value between the vertical resolution of the NTSC and PAL video formats. The interpolator module performs interpolation using a set of combined filters, each of the combined filters incorporating a demosaic filter and a scaling filter.

FIELD OF THE INVENTION

The invention relates to video image data processing, and moreparticularly to a video camera using a single image sensor to capturevideo images while providing video signals in multiple video imagestandards.

DESCRIPTION OF THE RELATED ART

Currently, NTSC and PAL are two television standards that are widelyused in the world. The NTSC (National Television Standards Committee)television standard, used primarily in the North America countries witha 60 Hz power grid, uses a frame rate of 30 frames per second (odd/evenfield interlaced) and 525 scan lines in a full frame of TV signals. ThePAL (Phase Alternate Lines) television standard, used primarily inEurope and Asia with a 50 Hz power grid, uses a frame rate of 25 framesper second (odd/even field interlaced) and 625 scan lines in a fullframe of TV signals.

FIG. 1 illustrates the raster scan operation of a television displayaccording to either the NTSC or PAL standard. Both the NTSC and PALstandards use interlacing where each frame of a television image isdisplayed in an odd field and an even field such that alternate lines ofthe full frame image is displayed in each field. A full frame videoimage includes a horizontal blanking region and a vertical blankingregion. Under the CCIR 601 standard, the NTSC video format has up to 858pixels in each of 525 scan lines, but only 720 active pixels in thehorizontal direction and 480 active pixels in the vertical direction togive the 4:3 aspect ratio of the television display. On the other hand,in the PAL video format, although there are up to 864 pixels in each of625 scan lines, there are only 720 active pixels in the horizontaldirection and 576 active pixels in the vertical direction. Otherstandards, such as those promulgated by the InternationalTelecommunication Union (ITU), may have other specification for scanlines and active pixels for the NTSC or PAL video format.

The differences in field rate, scan rate and other specification of thedifferent television standards give rise to incompatibilities in videoformats such that video recording equipments or video display equipmentsare typically manufactured dedicated to a specific standard. Videoimages recorded using one television standard (or video format) cannotbe displayed on viewing equipment supporting another standard withoutfirst converting the video recordings to the other standard.

Standard converters used to convert recordings from an original videoformat to a destination video format are known. These standardconverters are technically complex and expensive. For example,computationally intensive motion estimation algorithms are often used tointerpolate video image data between frames of image data in order togenerate final images having smooth motion. Standard converters aremostly used only by television broadcast stations to convert broadcastsignals from foreign countries to the television standard of the localcountry. Multi-standard VCRs and televisions are also known. Thesemulti-standard machines operate to display the video images in the videoformat in which the images were recorded and do not perform anyconversion of the images.

Conventional video cameras are typically manufactured for a specific TVstandard. This is primarily because the different numbers of scan linesper frame in the different video standards dictate differently shapedpixels for each standard. For example, under the CCIR 601 standard fordigital standard television signals, the aspect ratio for pixels in thePAL format is 1:0.94 to provide 720:576 active pixels in a full frameimage. On the other hand, the aspect ratio for pixels in the NTSC formatis 1:1.125 to provide 720:480 active pixels in a full frame image. Inconventional video cameras, separate image sensors are developed for theNTSC and the PAL standards to accommodate the different aspect ratiosrequired for the pixels. Thus, conventional video cameras are dedicatedequipment and support only one television standard in recording anddisplay.

A camcorder/still image camera that uses a high resolution image sensor(e.g. greater than 1 Megapixels) to record images and down-sample therecorded signals vertically and horizontally to generate video images ineither the PAL or the NTSC standard is known. However, this solution isundesirable because the high resolution image sensor can be very costlyand the down-sample processing can require large memory space toimplement. Also, because size of the image sensor determines the size ofthe optics to be used, a high resolution image sensor would requireoptics having larger sizes than the sizes of commercially availableoptics. The requirements for custom-made optics increases the cost ofthe camera. Alternately, if smaller pixels are used in the highresolution image sensor to limit the size of the sensor, the sensitivityof the image sensor is also lowered.

Therefore, it is desirable for a video image capture device capable ofgenerating video image data compatible with a number of video standards.

SUMMARY OF THE INVENTION

According to one embodiment of the present invention, a video imagecapture device includes an image sensor, a frame buffer, and aninterpolator module. The image sensor includes a two-dimensional arrayof pixel elements overlaid with a pattern of f selectively transmissivefilters and has a first vertical resolution different from the verticalresolutions specified for a group of video formats. The image sensoroutputs digital pixel data representing an image of a scene. The framebuffer, which is in communication with the image sensor, stores thedigital pixel data. The interpolator module, which is in communicationwith the frame buffer, interpolates the digital pixel data to generatevideo data in at least three color planes and having a second verticalresolution corresponding to a video format selected from the group ofvideo formats.

In one embodiment, the group of video formats includes the NTSC and PALvideo formats. The vertical resolution of the image sensor has a valuebetween the vertical resolution of the NTSC and PAL video formats.

In one embodiment, the interpolator module performs interpolation usinga set of demosaic filters and a set of scaling filters. In anotherembodiment, the interpolator module performs interpolation using a setof combined filters, each of the combined filters incorporating ademosaic filter and a scaling filter.

The present invention is better understood upon consideration of thedetailed description below and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the raster scan operation of a television displayaccording to either the NTSC or PAL standard.

FIG. 2 is a block diagram of the video image capture device according toone embodiment of the present invention.

FIG. 3 is a block diagram of a digital image sensor as described in U.S.Pat. No. 5,461,425 of Fowler et al.

FIG. 4 is a functional block diagram of an image sensor as described inU.S. patent application Ser. No. 09/567,786.

FIG. 5 is a detailed block diagram of a video image capture deviceaccording to one embodiment of the present invention.

FIG. 6 illustrates a color imaging array which can be used to implementthe sensor array in FIG. 5 according to one embodiment of the presentinvention.

FIG. 7 is a flow diagram illustrating the image data processing methodaccording to one embodiment of the present invention.

FIG. 8 is a flow diagram illustrating the image data processing methodaccording to an alternate embodiment of the present invention.

FIG. 9 is a flowchart illustrating the image data processing methodaccording to one embodiment of the present invention.

FIG. 10 is a flow chart illustrating the image data processing methodaccording to an alternate embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In accordance with the principles of the present invention, amulti-standard video image capture device uses a single image sensor andapplies vertical interpolation to generate video images in multiplevideo formats. In one embodiment, the video image capture device uses animage sensor having square pixels, that is, each pixel has a 1:1 aspectratio. The video image capture device captures video images having anintermediate vertical resolution and applies upsampling or downsamplingin the vertical direction to generate video signals in the desired videoformat.

System Overview

FIG. 2 is a block diagram of a video image capture device according toone embodiment of the present invention. Referring to FIG. 2, videoimage capture device 100 collects visual information in the form oflight intensity values using an image sensor 102. Image sensor 102 is anarea image sensor and includes a two-dimensional array of lightdetecting elements, also called photodetectors. Image sensor 102collects image data under the control of a processor 110. At apredefined frame rate, image data collected by image sensor 102 are readout of the photodetectors and stored in a frame buffer 104. Typically,frame buffer 104 includes enough memory space to store at least aportion of a frame of image data from image sensor 102. Frame buffer 104may also include memory allocation (such as memory space 124) forstoring the instructions used by processor 110.

Depending on the video format selected, the image data stored in framebuffer 104 is processed into video data in the desired video formatthrough the operation of an interpolator module 106. The desired videoformat can be selected in a variety of ways, such as by accepting aninput from the user through a mode select signal.

Interpolator module 106 performs vertical interpolation of the imagedata and either upsamples or downsamples to generate video data havingthe desired vertical resolution. For color applications, interpolatormodule 106 also performs color interpolation (“demosaicing”) to generatefull color video data. In one embodiment of the present invention,interpolator module 106 performs demosaicing and vertical interpolationin separate operations. In another embodiment of the present invention,interpolator module 106 performs both the demosaicing and verticalinterpolation operations in one combined operation, thereby reducing thecomputational burden and time required to process the image data. Thefull color video data in the selected video format are then provided toa TV encoder 108 to be encoded as video signals (or TV signals) for theselected television standard. TV encoder 108 can encode video data intoanalog or digital television signals (or video signals).

The encoded video signals can be used in any number of ways depending onthe application. For example, the signals can be provided to atelevision set 112 for display. The encoded TV signals can also be fedto a video recording device to be recorded on a video recording medium.When video image capture device 100 is a video camcorder, the TV signalscan be provided to a viewfinder on the camcorder.

In operation, TV encoder 108 drives video image capture device 100backward by transmitting control signals to interpolator module 106specifying the line number and the display field (odd or even) for whichvideo signals are to be processed. In response, interpolator module 106generates full color video data in the selected format for the linenumber and the display field specified. For example, when the NTSCstandard is selected, interpolator module 106 will generate video datahaving a vertical resolution of 240 lines per field. When the PALstandard is selected, interpolator module 106 will generate video datahaving a vertical resolution of 288 lines per field.

In the present description, video image capture device 100 generatesvideo signals in either the NTSC video format or the PAL video format.However, this is illustrative only and in other embodiments, video imagecapture device 100 can be configured to support any video formats andany number of video formats, as long as interpolator module 106 isprovided with the appropriate interpolation coefficients, as will bedescribed in more detail below.

The video image capture device of the present invention offers numerousadvantages not realized in conventional imaging devices. First, thevideo image capture device of the present invention providesmulti-standard capability, allowing a single imaging device to be usedto capture video images to be displayed in any number of televisionstandards. In essence, a user of the video image capture device of thepresent invention can capture video images and display or record theimages in any or all of the television standards. The multi-standardcapability of the video image capture device of the present inventionprovides convenience and ease of use not attainable in conventionalvideo imaging devices. The video image capture device of the presentinvention uses multiple lines of pixel data to create each scan line ofvideo images. The image capture device is capable of generating videoimages having adequate image quality, comparable with video imagescaptured using imaging devices with dedicated image sensors for thespecific video format (NTSC or PAL).

Second, the video image capture device captures video images having anintermediate vertical resolution and performs either upsampling ordownsampling to obtain the desired video format. Thus, the video imagecapture device of the present invention avoids the use of a costly highresolution image sensor.

Third, by using an image sensor having intermediate resolution, thevideo image capture device of the present invention also avoids the needfor a large memory buffer to store large amount of pixel data resultedfrom using a high resolution image sensor. In sum, the video imagecapture device of the present invention provides a cost effectivealternative in providing multi-standard capability.

Lastly, by using a single image sensor having a predefined resolutionregardless of the ultimate video format, image processing functions,such as temporal filters for noise reduction, can be greatly simplifiedas such image processing functions are performed on image data havingfixed input resolution, prior to the image data being processed into thespecific video format.

The detail structure and operation of video image capture device 100will now be described with reference to the detailed block diagram ofthe video image capture device in FIG. 5, a color imaging array in FIG.6, the process flow diagrams in FIGS. 7 and 8, and the flowcharts inFIGS. 9 and 10.

Image Sensor

As described above, video image capture device 100 uses a single imagesensor to capture video images which are then processed into video datain any video formats. Image sensor 102 of device 100 is an area imagesensor including a two-dimensional array of light detecting elements,also called photodetectors. Image sensor 102 may also include othercircuitry to support the operation of the image sensor for collectingimage data and reading out the image data from the array. For example,image sensor 102 may include control circuitry, such as addressdecoders, for accessing the array of photodetectors, readout circuitsfor reading out the pixel data from the array, analog-to-digitalconversion circuit for digitizing the pixel data, and a local memory(optional) for storing the pixel data.

In the present embodiment, image sensor 102 is a digital image sensorand can be implemented as a Complementary Metal-Oxide Semiconductor(CMOS) image sensor, such as an active pixel sensor (APS) or a digitalpixel sensor (DPS). Of course, image sensor 102 can be implemented usingany image sensor technology, presently available or to be developed. Ina preferred embodiment of the present invention, image sensor 102 isimplemented as a digital pixel sensor. A CMOS image sensor with pixellevel analog-to-digital conversion is described in U.S. Pat. No.5,461,425 of B. Fowler et al. (the '425 patent), which patent isincorporated herein by reference in its entirety. A digital pixel sensorprovides a digital output signal at each pixel element representing thelight intensity value detected by that pixel element. The combination ofa photodetector and an analog-to-digital (A/D) converter in an areaimage sensor helps enhance detection accuracy, reduce power consumption,and improves overall system performance.

In the present description, a digital pixel sensor (DPS) array or asensor array refers to a digital image sensor having an array ofphotodetectors where each photodetector produces a digital outputsignal. In one embodiment of the present invention, the DPS arrayimplements the digital pixel sensor architecture illustrated in FIG. 3and described in the aforementioned '425 patent. The DPS array of the'425 patent utilizes pixel level analog-to-digital conversion to providea digital output signal at each pixel. The pixels of a DPS array aresometimes referred to as a “sensor pixel” or a “sensor element” or a“digital pixel,” which terms are used to indicate that each of thephotodetectors of a DPS array includes an analog-to-digital conversion(ADC) circuit, and is distinguishable from a conventional photodetectorwhich includes a photodetector and produces an analog signal. Thedigital output signals of a DPS array have advantages over theconventional analog signals in that the digital signals can be read outat a much higher speed. Of course, other schemes for implementing apixel level A/D conversion in an area image sensor may also be used inthe image sensor of the present invention.

In the digital pixel sensor architecture shown in FIG. 3, a dedicatedADC scheme is used. That is, each of pixel element 14 in sensor array 12includes a ADC circuit. The image sensor of the present invention canemploy other DPS architectures, including a shared ADC scheme. In theshared ADC scheme, instead of providing a dedicated ADC circuit to eachphotodetector in a sensor array, an ADC circuit is shared among a groupof neighboring photodetectors. For example, in one embodiment, fourneighboring photodetectors may share one ADC circuit situated in thecenter of the four photodetectors. The ADC circuit performs A/Dconversion of the output voltage signal from each photodetectors bymultiplexing between the four photodetectors. The shared ADCarchitecture retains all the benefits of a pixel level analog-to-digitalconversion while providing the advantages of using a much smallercircuit area, thus reducing manufacturing cost and improving yield.

In the preferred embodiment of the present invention, the ADC circuit ofeach digital pixel or each group of digital pixel is implemented usingthe Multi-Channel Bit Serial (MCBS) analog-to-digital conversiontechnique described in U.S. Pat. No. 5,801,657 of B. Fowler et al. (the'657 patent), which patent is incorporated herein by reference in itsentirety. The MCBS ADC technique of the '657 patent can significantlyimprove the overall system performance while minimizing the size of theADC circuit. Furthermore, as described in the '657 patent, an MCBS ADChas many advantages applicable to image acquisition and moreimportantly, facilitates high-speed readout.

Although image sensor 102 of video image capture device 100 can beimplemented as any types of digital imaging device, the use of a DPS inimage sensor 102 has advantages over other imaging device in that a verylarge dynamic range in image capture can be achieved. More importantly,the high dynamic range image can be maintained throughout theinterpolation process such that the resultant video data can have a highdynamic range, regardless of the video format.

Copending and commonly assigned U.S. patent application Ser. No.09/567,638, entitled “Integrated Digital Pixel Sensor Having a SensingArea and a Digital Memory Area” of David Yang et al., describes anintegrated DPS sensor with an on-chip memory for storing at least oneframe of pixel data. The incorporation of an on-chip memory in a DPSsensor alleviates the data transmission bottleneck problem associatedwith the use of an off-chip memory for storage of the pixel data. Inparticular, the integration of a memory with a DPS sensor makes feasiblethe use of multiple sampling for improving the quality of the capturedimages. Multiple sampling is a technique capable of achieving a widedynamic range in an image sensor without many of the disadvantagesassociated with other dynamic range enhancement techniques, such asdegradation in signal-to-noise ratio and increased implementationcomplexity. Copending and commonly assigned U.S. patent application Ser.No. 09/567,786, entitled “Multiple Sampling via a Time-indexed Method toAchieve Wide Dynamic Ranges” of David Yang et al., describes a methodfor facilitating image multiple sampling using a time-indexed approach.The aforementioned patent applications are incorporated herein byreference in their entireties.

FIG. 4 duplicates FIG. 3 of the '786 patent application and shows afunctional block diagram of an image sensor 300 which may be used topractice the method of the present invention. The operation of imagesensor 300 using multiple sampling is described in detail in the '786patent application. Image sensor 300 includes a DPS sensor array 302which has an N by M array of pixel elements. Sensor array 302 employseither the dedicated ADC scheme or the shared ADC scheme andincorporates pixel level analog-to-digital conversion. A sense amplifierand latch circuit 304 is coupled to sensor array 302 to facilitate thereadout of digital signals from sensor array 302. The digital signals(also referred to as digital pixel data) are stored in digital pixeldata memory 310. To support multiple sampling, image sensor 300 alsoincludes a threshold memory 306 and a time index memory 308 coupled tosensor array 302. Threshold memory 306 stores information of each pixelindicating whether the light intensity value measured by each pixel insensor array 302 has passed a predetermined threshold level. Theexposure time indicating when the light intensity measured by each pixelhas passed the threshold level is stored in time index memory 308. As aresult of this memory configuration, each pixel element in sensor array302 can be individually time-stamped by threshold memory 306 and timeindex memory 308 and stored in digital pixel data memory 310. In thepresent embodiment, image sensor 102 is a DPS image sensor and isimplemented in the same manner as image sensor 300 of FIG. 4 to supportmultiple sampling for attaining a high dynamic range in image capture.

FIG. 5 is a detailed block diagram of a video image capture deviceaccording to one embodiment of the present invention. Like elements inFIG. 2 and FIG. 5 are given like reference numerals to simplify thediscussion. Referring to FIG. 5, image sensor 102 includes a sensorarray 210 of light detecting elements (also called pixels) and generatesdigital pixel data as output signals at each pixel location. Imagesensor 102 also includes an image buffer 212 for storing at least oneframe of digital pixel data from sensor array 210 and a data processor214 for performing image processing operations, such as normalization ofpixel data captured using multiple sampling.

Image sensor 102 may also include other circuits to support the imagingoperations. For instance, image sensor 102 may include a readout andcontrol circuit (not shown) for facilitating the readout process of theimage data captured by the sensor array. Image sensor 102 may alsoinclude row and column decoders, sense amplifiers and other controllogic (not shown). The digital pixel data (or image data) captured bysensor array 210 is read by the readout circuit and stored in imagebuffer 212 which is local (i.e., on the same integrated circuit) toimage sensor 102.

FIG. 6 illustrates a color imaging array which can be used to implementsensor array 210 according to one embodiment of the present invention.Referring to FIG. 6, sensor array 210 includes N rows and M columns ofphotodetectors. Thus, sensor array 210 has a resolution of N×M pixels.For color applications, sensor array 210 includes a mosaic ofselectively transmissive filters superimposed and in registration witheach of the photodetectors in the array so that multiple groups ofphotodetectors are made to sense different color spectrum of the visiblelight. In the present embodiment, sensor array 210 uses a “Bayerpattern” including individual luminance and chrominance sensingelements. In FIG. 6, sensor array 210 is implemented using a four-colorBayer pattern including green1 (G1) and green2 (G2) luminance sensingelements, and red (R) and blue (B) chrominance sensing elementsoverlaying a block of 2×2 pixels. The four-color Bayer pattern isrepeated throughout sensor array 210 so that each pixel is disposed tosample only one color component of the scene. Pixel values for othercolor components are missing at each pixel location. To obtain a fullcolor image, an interpolation process is performed amongst theneighboring pixels to determine the interpolated pixel values at eachpixel location for the missing color components. The color interpolationprocess in image sensor 100 will be described in more detail below.

In the present embodiment, in order to support both the NTSC and PALvideo format, sensor array 210 is configured to include 720 pixels inthe horizontal direction (i.e., 720 columns) and 540 pixels in thevertical direction (i.e., 540 rows). In the present embodiment, each ofthe pixels in image sensor 102 is a square pixel. That is, the pixels ofsensor array 210 each has a 1:1 aspect ratio. As thus configured, sensorarray 210 is well suited for television display which uses a 4:3 aspectratio.

Recall that for the NTSC video format, a full frame video image has 720active pixels in the horizontal direction and 525 active pixels in thevertical direction. On the other hand, for the PAL video format, a fullframe video image has 720 active pixels in the horizontal direction and625 active pixels in the vertical direction. Thus, in the presentembodiment, sensor array 210 is configured to have the same horizontalresolution as the NTSC and PAL video formats but an intermediatevertical resolution as compared to the NTSC and PAL video formats. Inthis manner, image data captured by sensor array 210 can be converted toeither the NTSC standard or the PAL standard by interpolating (orscaling) pixel data along the vertical direction only. Specifically,pixel data captured by sensor array 210, having a vertical resolution of540 pixels, is downsampled to obtain image data in the NTSC video format(240 lines per display field) (or the PAL video format (288 lines perdisplay field). Because no horizontal interpolation is needed, the imagequality of the final video images can be greatly improved. The operationof interpolator module 106 in upsampling or downsampling image data fromimage sensor 102 will be described in more detail below.

The 720×540 resolution of the image sensing array (array 210) selectedfor the present embodiment is illustrative only. In other embodiments,the image sensing array can have other resolution suitable for thetelevision standards to be supported. Also, the resolution of the imagesensing array can be selected to maintain compatibility with existing,cost effective optical systems. For example, in the present embodiment,the 720×540 resolution in combination with the specific pixel size ofsensor array 210 results in an image sensor with a 6 mm diagonaldimension which is compatible with existing optical systems.

Image Data Processing Method Overview

As described above, the image data captured by image sensor 102 isprocessed by interpolator module 106 into video data having the selectedvideo format. The image data processing operation includes two maincomponents. The first component is a color interpolation process toreconstruct missing pixel values at each pixel location using pixel datacaptured using a color filter pattern. The second component is avertical interpolation process where the image data, having a verticalresolution defined by the image sensing device (such as 540 lines perframe), are resampled into video data having the vertical resolutionspecified by the selected television standard (such as 240 or 288 linesper even/odd display field). FIGS. 7 and 8 are process flow diagramsillustrating the image data processing operation according to twoembodiments of the present invention.

Image data captured by image sensor 210 can be represented as atwo-dimensional array of pixel data, each pixel data associated with onecolor component of the four-color Bayer pattern. However, full colorvideo data are represented by three sets of pixel values, one set ofpixel value for each of the three primary color planes (such as red,green and blue) at every pixel location. Color interpolation is aprocess for deriving the pixel values for the missing color componentsat each pixel location using a neighborhood of pixels. Techniques forperforming color interpolation (“demosaicing”) are known in the art.U.S. Pat. Nos. 4,642,678 to Cok, 5,373,322 to Laroche et al., and5,475,769 to Wober et al. describe various methods for recoveringmissing pixel values from sampled color image data. The colorinterpolation process in the image data processing method of the presentinvention can apply any of the techniques described in the abovereferenced patents which patents are incorporated herein by reference intheir entireties.

In the present embodiment of the present invention, the colorinterpolation process in the image data processing method uses an n×nconvolution kernel to compute missing pixel values for each pixellocation based on the pixel values of a neighborhood of pixelssurrounding each pixel location. For an n×n neighborhood of pixels, theconvolution kernel is an n×n set of coefficients. In the interpolationprocess, a different kernel of coefficients is used for each color planeof the final full color image and for each color filter type in thefilter pattern. The use of convolution kernels in color interpolation isdescribed in the aforementioned Wober patent and also in copending andcommonly assigned U.S. patent application Ser. No. 10/006,974, entitled“Method of Defining Coefficients For Use In Interpolating Pixel Values,”of Benjamin P. Olding and Ricardo J. Motta, filed Dec. 5, 2001. The '974patent application is incorporated herein by reference in its entirety.In brief, full color pixel data are reconstructed by applying theappropriate convolution kernel to each pixel location of the sensorarray and computing the full color pixel values for each pixel location.

The color interpolation process requires a set of coefficients to bedetermined for each color plane in the full color image and for eachcolor filter of the filter pattern used by the image sensor. Thus, whenthe final full color image has three color planes (e.g., RGB) and thefilter pattern applied to the image sensor has four types of colorfilters (e.g., R, G1, G2 and B), a total of 12 n×n convolution kernelsare needed to interpolate the pixel data. In the present embodiment, thecolor kernel used for color interpolation is a square matrix (n×n).However, the use of a square convolution kernel in the presentdescription is illustrative only. In other embodiments, an n×n′convolution kernel, where n≠n′, can be used.

In one embodiment of the present invention, the method for definingcoefficients described in the aforementioned Wober patent is used. Woberdescribes a method for determining a set of weighting coefficients usinga linear, minimum mean square error solution of a matrix and vectorexpression. The matrix and vector expression defines the relationshipbetween a neighborhood of pixel values and the actual values of the sameneighborhood of pixels for a particular color component. In a preferredembodiment of the present invention, the method for definingcoefficients described in the aforementioned '974 patent application isapplied. In the '974 patent application, the coefficients for theconvolution kernels are computed by applying a constraint matrixspecifying one or more constraints. The constraints are selected toenhance the image quality of the resultant full color image, such ascolor uniformity and edge uniformity in the final image.

As described above, video image capture device 100 includes sensor array210 having a horizontal resolution that commensurate with the televisionstandards (e.g. NTSC and PAL) that device 100 is to support. Thus,device 100 performs only vertical interpolation to resample the imagedata into the resolution of the desired video format. Techniques forperforming vertical interpolation of pixel data are known. For example,U.S. Pat. No. 5,764,238, to Lum et al., and U.S. Pat. No. 6,347,154, toKaranovic et al., describe methods for scaling image data to bedisplayed in either the horizontal or the vertical direction. Theaforementioned patents are incorporated herein by reference in theirentireties.

FIG. 7 is a flow diagram illustrating the image data processing methodaccording to one embodiment of the present invention. Referring to FIG.7, a pixel array 270 represents a portion of sensor array 210 overlaidwith a Bayer filter pattern including four types of color filters. Pixeldata at each pixel location represents the intensity of the specificcolor component of the light impinging upon that pixel location. In thepresent embodiment, the image data processing method first applies colorinterpolation to reconstruct the missing color components so that imagedata representing a full color image is obtained. In the presentembodiment, full color video data in device 100 is represented in theRGB color space. The color interpolation process applies a set ofconvolution kernels (“demosaic filters”) to the image data to generateimage data in each of the RGB color planes, depicted as red color plane272, green color plane 274 and blue color plane 276 in FIG. 7. Note thatthe use of the RGB color space in the present embodiment is illustrativeonly. In other embodiments, the color interpolation process can beapplied to generate image data in any desired color space by using thecorresponding convolution kernels.

Then, the image data processing method applies vertical interpolation ofthe full color image data and provide video data having the desiredvertical resolution. For example, if the target video format is NTSC,vertical interpolation is applied to generate video data having avertical resolution of 240 lines per field. The vertical interpolationoperation applies a set of scaling filters (or scaling kernels) to thefull color image data where the scaling filters are functions of thehorizontal position of the scan line being processed.

FIG. 8 is a flow diagram illustrating the image data processing methodaccording to an alternate embodiment of the present invention. In theimage data processing method shown in FIG. 8, the color interpolationoperation and the vertical interpolation operation are combined by usinga combined demosaic/scaling filter. Thus, image data captured by sensorarray 210 (depicted by pixel array 270 in FIG. 8) are processed intofull color video data in the three color planes (278, 280 and 282)directly.

FIGS. 7 and 8 are provided to illustrate the two main components of theimage data processing operation of the present invention. Of course, theimage processing operation may include other signal processingfunctions, such as image enhancement operations, to enhance the qualityof the video image. In one embodiment of the present invention, theimage enhancement operation performs sharpening/softening of the imagedata by applying a frequency response correction filter to the imagedata. For example, the frequency response correction filter can be ahigh-pass filter to boost the high frequency response of the image data.In one embodiment, the image enhancement operation can be carried outseparately from the color interpolation and vertical interpolationoperations. In another embodiment, the image enhancement operation canbe combined with the color and vertical interpolation operations byintegrating the demosaic, scaling and frequency response correctionfilters into one combined filter.

Although the image data processing method can be operated by performingthe color interpolation, vertical interpolation and image enhancementoperations separately, certain advantages can be realized by combiningthe operations and applying one filter to process the image data in oneinterpolation operation. Specifically, when the image processingoperations are combined, there is no need to store intermediate resultsgenerated by each individual interpolation operation. Thus, the use of acombined filter conserves memory space required for the operation ofvideo image capture device 100, thereby conserving silicon resource whendevice 100 is manufactured as an integrated circuit. Additionally,applying a combined filter incorporating the color filter, the scalingfilter and the image enhancement filter (if any) eases the computationalburden by simplifying the image data processing operations and reducethe power consumption level of the video image capture device.

Image Data Processing Method

The detailed image data processing operation of video image capturedevice 100 of the present invention will now be described with referenceto the detailed block diagram of FIG. 5 and the flowchart of FIG. 9.FIG. 9 is a flowchart illustrating the image data processing methodaccording to one embodiment of the present invention.

At step 402 (FIG. 9), image sensor 102, operating under the control ofprocessor 110, captures image data by exposing sensor array 210 to thedesired scene or scenes of motion. Sensor array 210 is operated tosample the desired scene at a rate determined by the television standardselected. Thus, for NTSC standard, the image data is read out at a framerate of 30 Hz. For the PAL standard, the image data is read out at aframe rate of 25 Hz At each sampling, a full frame of image data, havinga resolution of 720×540 pixels, is read out and stored in image buffer212. The image data may be processed by data processor 214 (FIG. 5),such as to normalize image data captured using multiple sampling and toorganize the pixel data read out of the sensor array in the desiredorder.

At step 404 (FIG. 9), processor 110 directs the transfer of image datafrom image buffer 212 to frame buffer 104, such as through an interfacecircuit 120. Because large amount of image data is transferred fromimage buffer 212 to frame buffer 104, interface circuit 120 is a highspeed link. In one embodiment, interface circuit 120 uses low voltagedifferential signaling to facilitate the data transfer and can beoperated at up to 400 MHz.

Then, at step 406, interpolator module 106 reads the image data storedin frame buffer 104 and interpolates the image data to generate fullcolor video data (RGB) for the selected video format. Besides receivingthe image data from frame buffer 104, interpolator module 106 alsoreceives control signals from TV encoder 108. Specifically, TV encoder108 transmits control signals instructing interpolator module 106 whichline number in which display field (odd or even) to process image data.TV encoder 108 also transmits the mode select signal to interpolatormodule 106 instructing the module to generate video data in the desiredvideo format.

Image data from frame buffer 104 may be preprocessed beforeinterpolation is performed. Referring to FIG. 5, in the presentembodiment, image data from frame buffer 104 is first processed by a CDSand code conversion circuit 220. CDS or “Correlated Double Sampling” isa technique applied in image sensors for eliminating non-uniformity inthe sensor array. CDS can be used to correct for the variable comparatoroffset between the photodetectors in the array. When CDS is implemented,the sensor array is reset at the start of each capture. Then, thevoltage present at each of the photodetectors (also called the “CDSreset value” or “reset value”) is measured and stored in a designatedmemory location of the image sensor, such as image buffer 212.Subsequently, for each frame of pixel data captured by the sensor array,the stored reset values are subtracted from the corresponding pixelintensity value to derive the pixel data. In the present embodiment,circuit 220 is included to perform preprocessing of the image data bysubtracting the CDS reset values from the image data and converting theimage data represented in Gray code to binary representation.

In operation, interpolator module 106 loads portions of image data to beprocessed from frame buffer 104 into a line buffer 222. In the presentembodiment, line buffer 222 stores 9 rows of image data, that is imagedata for 6480 pixels are stored in line buffer 222. The size of linebuffer 222 is dictated by the size of the convolution kernels used forcolor interpolation and the size of the scaling filters used for thevertical interpolation process. The 5 by 9 configuration chosen in thepresent embodiment relates to the size of the combined convolutionkernel used to interpolate the image data, as will be described in moredetail below. In other embodiments, line buffer 222 may have otherconfigurations.

A 2-D interpolator 230 in interpolator module 106 reads the image datastored in line buffer 222 and performs the interpolation operations. Asdescribed above, interpolation of the image data can be performed in oneof two ways. In the first embodiment, the color interpolation andvertical interpolation operations are performed separately, asillustrated in FIG. 7. Interpolator 230 first performs colorinterpolation by applying the appropriate demosaic filter to the imagedata stored in line buffer 222. The coefficients for the demosaic filterare provided to interpolator 230 by a coefficient engine 224. Then,interpolator 230 performs vertical interpolation of the full color imagedata by applying the appropriate scaling filter to the full color imagedata. The scaling filter operates to scale the 720×540 resolution imagedata to the selected video format. For example, when the NTSC videoformat is selected, the scaling filter is applied to scale down theimage data to a resolution of 720×240 pixels per field. The coefficientsof the scaling filter are also provided by coefficients engine 224.Interpolator 230 can also apply image enhancement operations, such assharpening/softening, to the image data after the vertical interpolationprocess. In this embodiment, the coefficients for the demosaic filtersand the scaling filters can be stored in a coefficient memory 226 incoefficient engine 224. Coefficient engine 224 retrieves the desiredfilters by indexing memory locations in coefficient memory 226.

In the second embodiment of the present invention, the colorinterpolation and vertical interpolation operations are combined andperformed in one step, as illustrated in FIG. 8. In the presentembodiment, the color interpolation and vertical interpolationoperations are combined by combining the demosaic filter with thecorresponding scaling filter so that a combined filter can be applied tothe image data to generate video data in full color and having thedesired video format directly.

The color interpolation process applies a demosaic filter which is ann×n convolution kernel. In the present embodiment, a 5×5 convolutionkernel is used. To perform color interpolation, a total of 12convolution kernels are used in the present embodiment, one for eachcolor space (RGB) and for each filter type of the filter pattern (R, G1,G2 and B).

The vertical interpolation process applies a scaling filter which in thepresent embodiment, is a 1×5 scaling filter. In the present embodiment,a 5-tap filter is used to improve the high frequency response of thefilter operation. In other embodiments, an m-tap filter can be used toobtain the desired resolution. Increasing m (the number of taps) of thescaling filter improves the resolution of the final image. In theory, adifferent scaling filter is needed for each scan line of the video imagebecause the vertical interpolation is not always performed along pixelboundaries. For example, in the present embodiment, sensor array 210 has540 lines of pixels and thus each pixel is 1/540 in unit height.However, an NTSC display, for example, has 480 active lines and thuseach line has an unit height of 1/480. Therefore, vertical interpolationof each line in the NTSC display makes use of a different set of pixeldata from the image sensor. As a result, to attain infinite precisionand a perfect image, a large number of filters may be required. For NTSCformat, 72 different kernels are required. For PAL format, 240 kernelsare required.

However, in practice, only a limited number of scaling filters is neededbecause each scan line in a television display has limited precision.The limited precision of each scan line in a display is a result ofseveral factors. First, the frequency response of the transmissionchannel limits the spatial resolution in horizontal direction. Since thespatial resolution in the horizontal direction is limited already, thereis no need to be infinitely precise in vertical direction. Second,jitters in the deflection circuit of the display cause uncertainty invertical position of the scanning e-beam. Finally, the human visualperception capability will not be sufficient to notice a limitedaccuracy in the vertical or horizontal positioning of interpolatedpixel. The limited precision of each scan line can be exploited toreduce the number of scaling filters needed to obtain satisfactory imageresults. In the present embodiment, only thirty two scaling filters, S0to S31, are used for all of the scan lines in each of the video format.

In accordance with the present invention, the combined demosaic andscaling filter (“DS filter”) is given as follows:[D]×[S][DS],  Eq. (1)where matrix D is the 5×5 demosaic filter, matrix S is the 1×5 scalingfilter and matrix DS is the combined demosaic and scaling filter. In thepresent embodiment, the DS matrix is a 5×9 matrix generated by applyingthe scaling filter to each vertical position of the demosaic filter. Inthe present embodiment, 12 demosaic filters and 32 scaling filters areused to support the interpolation operations in interpolator 230. Thus,a total of 12×32 or 384 5×9 DS filters are needed to interpolate pixeldata for all of the pixels in sensor array 210.

In one embodiment of the present invention, the 384 DS filters can beprecomputed and stored in coefficient memory 226 of coefficient engine224. Interpolator 230 retrieves the DS filter it needs for theprocessing of image data for each scan line of the displayed image.However, the storage of the large number of pre-computed DS filtersrequires a large amount of memory and may not be desirable when device100 is manufactured in an integrated circuit where silicon real estateis critical. Furthermore, the memory access time required to retrieveeach DS filter from a large memory may not be fast enough to supportreal-time video display. Cache memory or other fast memory needs to beused to store the kernels so that the kernels can be accessed forreal-time display purposes.

Thus, in another embodiment of the present invention, the DS filters arenot precomputed and stored. Instead, only the 12 5×5 demosaic filters (Dfilters) and the 32 1×5 scaling filters (S filters) are stored and theDS filters that are required for each scan line of the display image aregenerated on-the-fly. Thus, in this embodiment, only 12+32 or 44 “raw”kernels (filters), instead of 384 filters, are stored in device 100,thereby significantly reducing the size of the memory required andconsequently reducing the memory access time required to address the rawfilters. Referring to FIG. 5, the raw filters (Di and Sj) are stored inan allocated memory location 228 in device 100. Coefficient engine 224operates to generate the DS filters necessary for each scan line basedon the control signals received from TV encoder 108, which controlsignals specifying the field number (odd/even) and the scan line numberfor which video data are to be generated.

In the present embodiment, the DS filters are generated during thehorizontal blanking time of the video display. Coefficient engine 224index raw kernel memory 228 to retrieve the raw filters. Coefficientengine 224 computes the DS filters based on the raw filters and storethe DS filters in coefficient memory 226 to be used for the processingof video data in the current scan line. For each scan line in thedisplay image of the video display, 6 DS filters are computed. 6 DSfilters are required because there can be at most two color filter typeson each line of the sensor array (see FIG. 6), thus 6 demosaic filtersare needed for each scan line. As a result, 3×2 or 6 DS filters arecomputed for each line of image data.

The 12 DS filters are used to interpolate image data in each row of thesensor array. That is, each set of 12 DS filters is used to interpolateimage data associated with 720 pixels on each row. As each row of imagedata is processed, image data in line buffer 222 is shifted and newimage data from frame buffer 104 is loaded into line buffer 222 so thatat any time 9 rows of image data are stored in the line buffer.Interpolator 230 interpolates image data from 45 (9×5) pixels, centeredaround the current pixel position and stored in line buffer 222 togenerate video data in the three color planes (RGB). The 5 columns ofimage data are processed to generate video data in the three colorcomponents. When interpolator 230 completes the interpolation of imagedata for one scan line of video data, interpolator 230 proceeds to thenext scan line of image data. In the present embodiment, the 6 DSfilters required for the next row of image data are generated during thehorizontal blanking time. In other embodiments, the next set of DSfilters can be generated concurrently with the processing of pixel datafor the current line as long as sufficient memory is provided to storethe next set of DS filters.

Returning to FIG. 9, interpolator module 106 processes the image dataand outputs video data for one field (odd or even) in the selected videoformat (i.e., 240 lines for NTSC standard or 288 lines for the PALstandard) (step 408 in FIG. 9). In the present embodiment, the videodata are provided to a tone correction module 126 (FIG. 5), a CSC (ColorSpace Conversion) module 128 (FIG. 5) and an enhancement filter module130 (FIG. 5) before being encoded into video signals into the selectedformat (NTSC or PAL). In one embodiment, enhancement filter module 130includes a frequency response correction filter to perform imagesharpening and softening. In another embodiment, enhancement filtermodule 130 can also include a set of median filters for noise reduction.

Television images are displayed in odd and even field interlaced format.Thus, interpolator module 106 processes image data to generate videodata for one display field at a time. In the present embodiment, imagedata processing method generates video data in the odd and even fieldusing the same set of image data. Thus, at step 410, method 400determines if both display fields have been processed. If not, method400 returns to step 406 and interpolates the same frame of image datastored in frame buffer 104 to generate video data for the next displayfield.

FIG. 10 is a flow chart illustrating the image data processing methodaccording to an alternate embodiment of the present invention. FIG. 10provides an alternate means to generate video signals for the odd andeven display fields. Referring to FIG. 10, image data processing method500 operates by sampling image sensor 102 for new image data for eachdisplay field. Thus, at step 502, image data samples the desired sceneand image data is read out at a rate determined by the televisionstandard selected. Thus, for NTSC standard, the image data is read outat a field rate of 60 Hz. For the PAL standard, the image data is readout at a field rate of 50 Hz. Each frame of image data is stored inframe buffer 104 (step 504). Then, interpolator module 106 performsinterpolation of the image data as described above with reference toFIG. 9 (step 506). Interpolator module 106 generate video data for onedisplay field in the selected video format (step 508). Then, method 500returns to step 502 to read out another frame of image data from whichthe video data for the next display field will be derived. Interpolatormodule 106 determines which display field to generate video data forbased on the display field number signal received from TV Encoder 108.

The above detailed descriptions are provided to illustrate specificembodiments of the present invention and are not intended to belimiting. Numerous modifications and variations within the scope of thepresent invention are possible. For example, the resolution of theimaging array can be configured for the desired video formats.Accordingly, when HDTV, having an aspect ratio of 16:9, is applied, theimaging array can have the corresponding number of pixels in thevertical and horizontal direction. Furthermore, the video image capturedevice of the present invention can be implemented as an integratedcircuit or a set of integrated circuits. In one embodiment, image sensor102 is manufactured in one integrated circuit and the remainingcomponents of video image capture device 100 are manufactured in asecond integrated circuit. The present invention is defined by theappended claims.

1. A video image capture device, comprising: an image sensor comprisinga two-dimensional array of pixel elements overlaid with a pattern of agiven number of selectively transmissive filters and having a firstvertical resolution being an intermediate vertical resolution ofvertical resolutions specified for a group of video formats, said imagesensor outputting digital pixel data representing an image of a scene; aframe buffer, in communication with said image sensor, for storing saiddigital pixel data; and an interpolator module, in communication withsaid frame buffer, for interpolating said digital pixel data byupsampling or downsampling in the vertical direction only to generatevideo data in at least three color planes and having a second verticalresolution corresponding to a video format selected from said group ofvideo formats; wherein said interpolator module applies at least onecombined filter for interpolating said digital pixel data into videodata in a first color plane and having said second vertical resolution,said combined filter incorporating a demosaic filter and a scalingfilter, and wherein said demosaic filter is an n×n′ convolution kernelwhere n=n′ or n≠n′, said scaling filter is a 1×m convolution kernel, andsaid combined filter is an n×(n′+m−1) convolution kernel.
 2. The deviceof claim 1, wherein said group of video formats comprises NTSC and PALvideo formats.
 3. The device of claim 2, wherein said first verticalresolution is 540 pixels.
 4. The device of claim 1, wherein saidinterpolator module receives a mode select signal selecting said videoformat having said second vertical resolution from said group of videoformats.
 5. The device of claim 1, further comprising: a processor, incommunication with said image sensor, said frame buffer and saidinterpolator module, for directing said image sensor to output pixeldata, storing said pixel data in said frame buffer and operating saidinterpolator module to process said pixel data.
 6. The device of claim1, further comprising: a TV encoder, in communication with saidinterpolator module, for encoding said video data in said selected videoformat and for providing control signals to said interpolator module,said control signals identifying a display field and a scan line forwhich video data are to be processed.
 7. The device of claim 1, whereineach of said pixel elements of said image sensor generates analogsignals representative of said image, and said image sensor furthercomprises an analog-to-digital converter for converting said analogsignals to said digital pixel data.
 8. The device of claim 1, whereinsaid image sensor comprises a two-dimensional array of digital pixels,each of said digital pixels outputting digital signals as said digitalpixel data representative of said image.
 9. The device of claim 1,wherein said image sensor further comprises a data memory, incommunication with said array of pixel elements, for storing saiddigital pixel data generated by said array prior to transfer to saidframe buffer.
 10. The device of claim 1, wherein said interpolatormodule applies at least one demosaic filter for interpolating saiddigital pixel data into video data in a first color plane and applies atleast one scaling filter for interpolating said video data in said firstcolor plane into video data having said second vertical resolution. 11.The device of claim 1, wherein said interpolation module furthercomprises a coefficient memory for storing a set of precomputed combinedfilters.
 12. The device of claim 1, wherein said interpolation modulefurther comprises a coefficient engine and a raw kernel memory storing aset of demosaic filters and a set of scaling filters, said coefficientengine computing said at least one combined filter momentarily prior tointerpolating said digital pixel data.
 13. The device of claim 12, saidvideo data is coupled to a display device for display and saidcoefficient engine computes said at least one combined filter during ahorizontal blanking time of said display device.
 14. The device of claim10, wherein said interpolation module further applies an imageenhancement filter to said video data, said image enhancement filtercomprising a frequency response correction filter.
 15. The device ofclaim 14, wherein said image enhancement filter comprises a set ofmedian filters for noise reduction.
 16. The device of claim 1, whereinsaid combined filter further incorporates an image enhancement filter,said image enhancement filter comprising a frequency response correctionfilter.
 17. The device of claim 16, wherein said image enhancementfilter comprises a set of median filters for noise reduction.
 18. Thedevice of claim 1, wherein said interpolator module comprises: aplurality of line buffers for storing a portion of said digital pixeldata to be interpolated; a raw kernel memory for storing a set ofdemosaic filters and a set of scaling filters; a coefficient engine forcomputing a plurality of combined filters using said set of demosaicfilters and said set of scaling filters, said plurality of combinedfilters being stored in a memory in said coefficient engine; and aninterpolator, in communication with said plurality of line buffers andsaid coefficient engine, for interpolating said digital pixel data usingsaid plurality of combined filters and generating video data in at leastthree color planes and having said second vertical resolution.
 19. Amethod for generating video signal, comprising: generating digital pixeldata representative of an image of a scene using an image sensor, saidimage sensor comprising a two-dimensional array of pixel elementsoverlaid with a pattern of a given number of selectively transmissivefilters and having a first vertical resolution being an intermediatevertical resolution of vertical resolutions specified for a group ofvideo formats; storing said digital pixel data in a frame buffer;processing said digital pixel data to generate video data in at leastthree color planes by interpolating said digital pixel data using aplurality of demosaic filters; in response to a select signal having afirst value selecting a first video format, processing said video databy upsampling or downsampling in the vertical direction to generatevideo data having a second vertical resolution associated with the firstvideo format by interpolating said video data using a plurality ofscaling filters; and in response to said select signal having a secondvalue selecting a second video format, processing said video data byupsampling or downsampling in the vertical direction to generate videodata having a third vertical resolution associated with the second videoformat by interpolating said video data using a plurality of scalingfilters; wherein said plurality of demosaic filters and said pluralityof scaling filters comprise at least one combined filter forinterpolating said digital pixel data into video data in a first colorplane and having said second or third vertical resolution said combinedfilter incorporating a demosaic filter and a scaling filter, and whereinsaid demosaic filter is an n×n′ convolution kernel where n=n′ or n≠n′said scaling filter is a 1×m convolution kernel and said combined filteris an n×(n′+m−1) convolution kernel.
 20. The method of claim 19, whereinsaid group of video formats comprises NTSC and PAL video formats. 21.The method of claim 19, wherein said first vertical resolution comprisesa value between said second vertical resolution and said third verticalresolution.
 22. The method of claim 19, further comprising: processingsaid video data using an image enhancement filter; and encoding saidvideo data as video signals in a corresponding one of said first andsecond video format.
 23. The method of claim 22, further comprising:after encoding said video signals for a first display field, returningto said processing said digital pixel data to generate video data in atleast three color planes to generate video data for a second displayfield different than said first display field.
 24. The method of claim22, further comprising: after encoding said video signals for a firstdisplay field, returning to said generating digital pixel datarepresentative of an image of a scene to generate video data for asecond display field different than said first display field.
 25. Amethod for generating video signal, comprising: generating digital pixeldata representative of an image of a scene using an image sensor, saidimage sensor comprising a two-dimensional array of pixel elementsoverlaid with a pattern of a given number of selectively transmissivefilters and having a first vertical resolution being an intermediatevertical resolution of vertical resolutions specified for a group ofvideo formats; storing said digital pixel data in a frame buffer; inresponse to a select signal having a first value selecting a first videoformat, processing said digital pixel data by upsampling or downsamplingin the vertical direction to generate video data in at least three colorplanes and having a second vertical resolution associated with the firstvideo format by interpolating said digital pixel data using a pluralityof combined filters, each of said combined filters incorporating ademosaic filter and a scaling filter; and in response to said selectsignal having a second value selecting a second video format, processingsaid digital pixel data by upsampling or downsampling in the verticaldirection to generate video data in at least three color planes andhaving a third vertical resolution associated with the second videoformat by interpolating said digital pixel data using a plurality ofcombined filters, each of said combined filters incorporating a demosaicfilter and a scaling filter; wherein said demosaic filter in each ofsaid combined filters is an n×n′ convolution kernel where n=n′ or n≠n′said scaling filter is a 1×m convolution kernel and said combined filteris an n×(n′+m−1) convolution kernel.
 26. The method of claim 25, whereinsaid group of video formats comprises NTSC and PAL video formats. 27.The method of claim 25, wherein said first vertical resolution comprisesa value between said second vertical resolution and said third verticalresolution.
 28. The method of claim 25, wherein each of said pluralityof combined filters further incorporates an image enhancement filter.29. The method of claim 25, further comprising: encoding said video dataas video signals in a corresponding one of said first and second videoformats.
 30. The method of claim 29, further comprising: after encodingsaid video signals for a first display field, returning to saidprocessing said digital pixel data to generate video data for a seconddisplay field different than said first display field.
 31. The method ofclaim 29, further comprising: after encoding said video signals for afirst display field, returning to said generating digital pixel datarepresentative of an image of a scene to generate video data for asecond display field different than said first display field.